Accessibility of Depicted Events Influences their Priority in Spoken Comprehension

Studies that monitor attention in depicted event scenes during utterance comprehension show that people prefer to rely on depicted events rather than their stereotypical knowledge. However, the presence of depicted events in the scene may have exaggerated their importance. Two eye-tracking experiments examined this issue by varying the accessibility of scene information. When we varied the visual presence of the scene (the scene disappeared before the utterance was heard), findings confirmed a greater relative priority of depicted events (Experiment 1). In contrast, when we altered the temporal extension of scene events (scene presentation emulated that they had been completed), people neither had a preference to rely on depicted events nor on their stereotypical knowledge (Experiment 2). These findings suggest that the visual presence of scene events cannot account for the preference to rely on depicted events. We discuss our findings in the context of research on the accessibility of events in discourse comprehension (e.g., Zwaan, Maaden, & Whitten, 2000). The priority of depicted events Prior research has revealed important insights into how we incrementally understand utterances that relate to a concurrently presented scene. In particular, the monitoring of eye movements in a scene while people listen to a concurrently presented and related utterance has shown that distinct types of information linguistic and world knowledge as well as depicted events are rapidly used for incremental thematic role assignment. This was revealed through the anticipation of relevant role fillers in the scene based on either linguistic and world knowledge (Kamide, Scheepers, & Altmann, 2003) or depicted agent-action-patient events (Knoeferle, Crocker, Scheepers, & Pickering, 2005) (see Tanenhaus, SpiveyKnowlton, Eberhard, & Sedivy, 1995, on the use of a visual referential context). To further compare the relative importance of verbmediated world knowledge about likely role-fillers with that of verb-mediated depicted events, Knoeferle and Crocker (accepted) carried out a study that directly compared the importance of these two informational sources. Participants in their study heard German object-verbsubject (OVS) sentences such as ‘The pilot (object) jinxes the wizard (subject)’ while inspecting the scene in Fig. 1. The noun phrase ‘the pilot’ established reference to the pilot in the scene. Object case marking on the determiner of the noun phrase together with the verb (‘jinx’) identified the pilot as the patient of a jinxingaction. The scene does not depict a jinxing action, but based on the verb ‘jinx’, we find a typical jinxing-agent (wizard) in the scene. In contrast, when we hear ‘The pilot (object) serves food to ...’, verb-based knowledge of stereotypical agents (e.g., a cook) cannot guide comprehension, as the scene affords no cook. However, the scene does afford a depicted food-serving event performed by the detective. Based on findings by Kamide et al. (2003) and Knoeferle et al. (2005), people should exploit either stereotypical knowledge of likely agents (‘jinx’) or depicted events (serving-food) for incremental thematic interpretation at the verb, and anticipate the appropriate agent. Figure 1: Example image for sentences in Table 1 In a further condition pair, people heard ‘The pilot (object) spies-on ...’. In this case, the scene affords both a stereotypical agent (the detective), and an immediately depicted agent of a spying event (the wizard) as relevant agents given the verb (see Fig 1). When people hear ‘spies-on’, word meaning, stereotypical knowledge of spy-on, and scene affordances allow them to anticipate either a depicted spying-event and its agent (the wizard), or a different, stereotypical agent (the detective). The comprehension system has to choose between two available, yet conflicting, types of information in online thematic role-assignment. Gaze pattern confirmed prior findings that people rely on either stereotypical knowledge (Kamide et al., 2003) or depicted events (Knoeferle et al., 2005) when the verb (‘jinx’, ‘serves-food-to’) uniquely identified a rel-